In order to solve the problem of action localization in video, a weakly supervised method based on template matching was proposed. Firstly, several candidate bounding boxes of the action subject position were given on each frame of the video, and then these candidate bounding boxes were connected in chronological order to form action proposals. Secondly, action templates were obtained from some frames of the training set video. Finally, the optimal model parameters were obtained after model training by using action proposals and action templates. In the experiments on UCF-sports dataset, the method has the accuracy of the action classification increased by 0.3 percentage points compared with TLSVM (Transfer Latent Support Vector Machine) method; when the overlapping threshold is 0.2, the method has the accuracy of action localization increased by 28.21 percentage points compared with CRANE method. Experimental results show that the proposed method can not only reduce the workload of dataset annotation, but also improve the accuracy of action classification and action localization.